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ABSTRACT 
An‘ approach for merging quantitative-qualitative data 
in order to enlarge the evaluators' perspective and provide an 
enriched data base for evaluating elusive evaluation problems in 

school settings is described. Mini-case studies were conducted over a 
period of three years in 11 urban schools to examine the impact of an , 
arts-in-the-schools program and tdentify the cluster of factors that 
impede or enhance successful*implementation and program effects. The 
replicable model for conducting mini-case studies and ore apes 
within and across school data illustrates that quantitative d 
qualitative data are integrative and serve a confirmatory purpose. 
Both qualitative and quantitative data have inherent limitations, 
however; integration of the two will extend the scope of the data 
base. Integration of the two types can generate new variables. 
Abandoning the view of quantitative and qualitative data as. 
dichotomous permits the development of synthesizing methods for 
discerning program effects in an organized, rational, scientific 
manner. (Author/GR) 
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The past two decades have witnessed both an intellectual and ideological 
struggle over what constitutes meaningful research methodology in the bahay) 
sciences. We need only read the opening two pages of many papers presented 
at AERA annual programs over the last several years to be cited a litany 


of the strengths and weaknesses of naturalistic versus experimental studies, 


“ED 204396: 


Quantitative versus qualitative methods, or soft versus hard research. The 
“old masters" are called up like heavy artillery to support a given position. 
One side or the other gets an additional bonus when it can reference a "hard" 
researcher going "soft" or vice versa: While some voices during the late 
1960's called for eclectic approaches to research (Weiss and Rein, 1969; Weiss, 
1970), these were lost in the noise of heated discussion. 

Today, however, researchers are facing up to the magnitude of establishing 
certainty; no one research method can provide the answers. Boruch and Wortman 
(1979) call for an inderstandting of how quantitative and qualitative methods 
can be "exploited jointly and more systematically." Maltz, Kerker and Gainey 
(1980) speak of accommodation: while epistomologies differ, they share common 
elements in an attempt to understand phenomena. Cronbach and Associates (1980) 
have this to say on the subject: 

"The evaluator wil] be wise not to declare allegiance to either \ 


quantitative-manipulative-summative methodology or a qualitative- 


zm 3/0 393 


naturalistic-descriptive methodology. He can draw on both styles 

at appropriate times and in appropriate amounts. Those who advo- 

cate an evaluation plan devoid of one kind of information or the 

other carry the burden oy justifying such exclusions." (p. 223) 
- ‘\ 


_*Paper presented at the Anerican Educational Research Association annual 
meeting, Division D, Los Angeles, California, April 198). 
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There is growing interest among evaluation researchers in the use of case 
- Studies to understand the social, organizational and political coftext of 
social programs. Case studies, many hasten to add, augment evaluation studies 
rather than stand alone. Lynn, dr. (1980) calls on the use of case methods 
in policy analysis to help vance with the context, but cautions analysts 
to use other analytical methods as well. Baltzell (1980) reflects on the need 
to integrate quantitative and qualitative methodologies to cient for 
shortcomings of both. She proposes a "standard case study" to facilitate the 
integration. Cronbach and Associates (1980) see the role of qualitative case 
studies within a statistical plan. Cautionary note is sounded by Boruch and 
Wortman (1970) if case studies are permitted to stand alone and not incorporated 
in a larger summative research design: 
“ay outcome evaluation which relies very heavily on these case re- 
parte: especially if they cannot be tied to the macro-level nin 
‘cesses underlying program success or failure, is unlikely to be 
much more than employment programs for the ethnographically inclined." 


(p. 319) 


The ProbTem 

The Office of Evaluation Research faced the task of conducting an evalua- a 
tion of an arts-in-the-schools program over_a three-year period. Obtaining 
quantitative data was not the problem, especially when word was handed down 
from higher administration to cooperate. But getting a handle:on an elusive 
program was another matter. Like most social programs conducted in large urban 
schools, identifying the program in situ presented difficulties. Without 


an understanding of what the program looked like from one school to the next, - 


and the contextual forces impinging on program implementation, the 


a 
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standard quantitative data would be uninterpretable. Qnce the evaluation 


err underway, it became clear to us that the textbook approach to program 
evaluation, that is, obtaining Student and teacher data on an array of 
variables ina pre-post/experimental-contro] design, wou}d. not illuminate | 
Depgran effects nor provide the decision maker with eres on how 

to adapt or modify the program should it be retained. We settled on the 
inclusion of mini case studies of each school as a methodological) vehicle for 
obtaining descriptive data. This approach differs from the standardized case 
study reported by Baltzell (1980) in that our instruments for collecting qual- 
itative data were more detailed than topic outlines but sufficiently flexible 
to permit reporting of events and perspectives not specifically fdentified 

in the instruments. Our problem, then, was to bring together the best of both 
worlds. We needed to find how to fnteorate the two worlds: (1) by fitting 
qualitative descriptions into a quasi-experimental design; (2) by using the 
qualitative data to confirm the quantitative finding; (3) by using both types 


of information dialectically, thereby creating a new synthesis. 


The Model Builders Project and the Schools 


The Model Builders Project (MBP) was a three-year arts-in-the schools pro- 
gram sponsored by a private, non-profit, urban-based organization. Whe purpose 
of the MBP was to integrate art-related) activities into the existing curriculum 
of urban, lower socioeconomic status schsols and thereby enrich the lives of 
students, teachers, parents, and the general community. At the inception of 
the MBP, five goals were farmitated to give direction to the program: 

1. To enable teachers to integrate the arts into the teaching of core 

curriculum, . 
2. To sustain and promote new avenues of communication between home, 


school, and community; 
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3. To enable children to use the arts as a resource for communication 
and self-realization, and as a means for sharpening perception and 
facilitating conceptual learning; 

4. To strengthen the. skills and understanding of the artist/instructors 
who plan and carry out the program; 

5. To continue research and the development of replicable models for 

| the integration of the arts into the total instructional program. 

Ten Chicago inner city schools with the following characteristics were 

selected to participate in the MBP: 1) serve a large minority population, 

2) situated in lower-income area of Chicago, 3) compfygted an application for . 
participation jn the MBP, 4) demonstrated a commitment of teachers, administra- 
tors, and parents to participate in MBP and its evaluations, and 5) agreed 

to provide necessary physical facilities and equipment. By concentrating 

the Project's resources in 10 sites,.the effects of a total. impact program 

could be tested without straining limited MBP resources. A total of 13 experi- 
mental schools (one site included a cluster of four schools) participated in 

~ the Program. The MBP continued in the 10 sites through Year 2. In the third 
year, the cluster site became an Academic Interest Center in the Chicago School 
System's Access to Excellence Program. In addition, in the third year one 
school site by notualvanreanent discontinued participation, leaving eight schools 
in the Project. 

Two matching schools, selected as comparison schools, were also avantied 

over the three year period, to serve as bench marks for measuring changes in 

the MBP zehgeie: One comparison school was viewed as a "model" inner city school, 
One that was said to function. The other had all the stereotypic problems 
associated with an inner city institution. Because these schools represented 


extremes rather than typical schools, they assumed two ends of an educational 


) 


‘ 
s 


continuum on which the MBP schools could be viewed. We found this way of 


representing a control in our evaJuation design to be a rich source Of compara- 


tive data. ‘ 
The key to the program were practicing artists who served as artist/in- 
structors. They could bring to teachers and the classes a vitality unemcum- 


bered by many of the rules and regulations of a large system. A number of. | 
program offerings made up the MBP. These included MBP re taught by artist/ 
instructors for teachers, other school staff, and spincipals: all-school assem- 
blies; .inservice workehane: artist/instructor "parlor meetings" with parents; 
artists/instructor visits in the classrooms ; city-wide principal meetings; 
city-wide teacher, principal, and artist/instructor HOPKINS and parent par- 

' ticipation in a city-wide,. community .council sponsored by the arts organiza- 
tion. 


® 
The Structure of the Mini-Case Studies 


The Office of Evaluation Research (OER) viewed its evaluation role through- 
out most of the three years as formative. Each year's evaluation design focused és 
on obtaining information about MBP's operations from several perspectives. 
In Year 1, a rather traditional pilot test design was used. Three groups of \ 
teachers and students were identified: 1) experimental (EXP): MBP participants, 
2) comparison 1(C-1): non-MBP participants in one of the 10 MBP sites, and 
3) comparison 2(C-2): non-MBP participants in one of the two comparison (non-MBP) 
schools. Self-reporting questionnaires, structured interviews, and structured 
classroom observations were completed to gather data permitting the following 
comparisons: 1) EXP, C-1 and C-2 classrooms on students' perception of their 


“classroom learning environment, 2) EXP and C-1 students on degree of program 


“. 


implementation, 3) EXP, C-1, C-2 teachers on art related activities, and 4) 
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EXP teachers and artist/instructors on course evaluations. Thus, the tenets 
of good quantitative research design were followed. However, when the data 
were analyzed, these measures discerned no project’ impact. Yet, when OER staff 
visited the schools, there were signs of project awareness and activities. 
There were also signs of external forces that impeded the functioning of the 
project. It was evident to OER staff that both internal and external occurrences 
impinging on the project were not being captured by traditional instrumenta- 
tion. OER concluded that the instruments were too tightly structured to capture 
what was really happening in the schools and to the Schools, Rather than blam- 
ing non-impact on external forces, OER,felt a need to gather more personal 
and detailed descriptions of each school. In order to obtain a clearer picture 
of how the project functioned under differing and common sets of constraints 
at different school sites, OER developed a mini-case study procedure. The 
mini-case studies were conducted in Years.2 and 3 to examine impact both within 
a given site and across all sites (in addition to Year 1 types of data). 
Obtaining approval to conduct case studies took some planning. At a meeting 
with all the "principals in the project, OER shared the first year's evaluation 
data, and some of the problems of getting at subtle types of data. Permission 
for three-member teams to have access to all] classrooms, all other educational 
supporting resources in the school and personnel was obtained. It took two 
very domnvited principals, however, to convince their colleagues that this 


wasn't a threatening act; rather, it would help improve the project and its 


implementation. 

Data for the mini-case studies were gathered from sl a albeit 
open-ended observation and interview instruments by trained observers and in- 
terviewers. The School Observation Form and interview forms were devised to 

‘ yield parallel data, yet were sufficiently flexible to permit gathering of 


“ data not falling into discrete, predictable categories. Thus each mini-case 
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study included comparative data that were also amenable to aggregation. The 
primary purpose of the instruments was to reach all C-1 staff who could in 
some way affect the schools' program and could give important perspectives 
on the functioning of the program. Thus, potential data sources included prin- 
ctpals, other school administrators, teachers, custodians, lunchroom personnel, 
Office staff, students, school community representative and parents, as well 
as the artist/instructors and the agency's staff. The School’ Observation Form 
was divided into nine sections: 1) physical characteristics, 2) social and 
inter-personal environment, 3) materials and equipment availability and utili- 
zation, 4) academic environment, 5) evidence of MBP-related activities (for 
the control schools, general arts-related activities were observed); 6) class- 
room environment, 7) behavior norms, 8) parents and visitors, and 9) overall 
impressions. For each section, semi-structured, open-ended questions were 
responded to (e.g.: describe sxanptes of social interactions you observe in 
the student lunchroom; describe the relationship of teachers to each other-- 
do they freely socialize or keep to themselves) and general, non-structured, 
open-ended impressions of OER staff were recorded (e.g., characterize the social 
environment; what is your overall impression of this school; how does the reward 
system work). The only restriction on the open-ended impressions was that 
all statements had to be supported by observable evidence (including statements 
by data sources). For example, if an observer stated: "Although there is little 
evidence of Model Builders activities in classrooms, there is indirect evidence 
’ of MBP impact," accompanying data to support that statement was required: "No 
MBP-related activities were observed in the classrooms. However, several teachers 
reported using drama in one form or another in their classes and in several 
classes, art projects from last year's MBP involvement were on display." (Data 


sources: classroom, hallways, teachers. ) 


. 
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The Pincipal Interview (PI) was divided into six sections: 1) MBP demands 
on the school's operation, 2) extent of principal involvements and support 
of MBP, 3) effectiveness of MBP activities in school, 4) observable Changes 
in academic or social environment that can be attributed t MBP, 5) extent 
of teacher and parent involvement in MBP, 6) external forces affecting MBP's 
implementation. Interviewers were instructed to cover 16 specific points in 
the interview; ‘aaieiale: probing questions, the order in which points were covered, 
and additjonal questions (based on what had been observed during same lwenee 
of the school observations) were left to the discretion of the interviewer. 
The interviewees were encouraged to document all statements with examples. 

School Observation Forms were completed independently by 3 observers 
who each spent an entire day at the school. Observers were given free reign 
by the principals to enter any and all classrooms and other school facilities 
(lunchroom, playgrounds, gyms, special-function rooms, etc.). Principal 
interviews were conducted by the cesignates leader of each three-person ob- 
servation team. The leader was one of three OER staff members involved with 
the MBP evaluation from its inception; in most cases a strong relationship 
of trust had been developed between the team leader and school principal. 
Semi-structured interviews with the school-community representatives, and 
informal interviews with custodians, parents in the building, office staff, ~ 
food service staff, faculty and students were additional sources of data. 
On completion of the observations and interviews, the three-member teams 
combined their information and impressions on a single observation form, 
from which the team leader wrote a draft summary case study and sybmitted t 
the draft to the other team members. Initially the plan was to enter the 
final approved case study as part. of the formal data bank in two forms: as 
self contained mini-case studies by school and as a single case study of 


, the total project. 
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Organizing, Quantifying, and Analyzing the Case Study Information 
OER had originally planned to organize the evaluation report into two 
sections: 1) quantitative-data and 2) supplementary qualitative data. The 
mini-case studies and composite case study would provide bh backaround for 
interpreting the quantitative analyses. As the case studies were being pre- 
pared, however, it became evident to OER staff that much of the qualitative 
data being gathered had suarilfiabieospeets that were relevant and important _ 
not only to a composite case study but as additions to the quantitative data 
bank. These patterns within the quaTitatiy, data were converted into quanti- 
tative form via categorization and ratings and added to the already-gathered 
quantitative data. Table 1 shows a draft form for summarizing descriptive 
© and Quantitative data in preparation for integrating the information. But 
the overlapping and combining of quantitative and qualitative data (and vice 
versa) did more than supplement wach other. Integration of data permitted 
OER staff ninbars to reach higher levels of understanding of interrela- 
tionships. In addition to the two data forms providing supportive information, 
a spiral effect was occurring; each type of information when combined displayed 
a dynamic interconnecteédness. The integration was leading to a synthesis 
which produced a new variable. For example, consider the variable of principal 
Support. . 
From the qualitative data gathered through principal interviews and the 
quantitative data from teacher and artist/instructor questionnaires, a quanti- 
tative rating of principal support. of the MBP was assigned (1 = minimal to 
4 = extensive). From MBP's archival attendance records, the quantitative 
_data of whether or not the principal attended the semester-long MBP course 
taught by the artist-instructors was determined (1 = no, 2 = yes). From 


the quantitative Course Evaluation Questionnaires completed by MBP teachers, 
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Table 1. Comparative Summary Table of Case Studies by School 
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Comparative ginoaes Table of Case Studies by School (continued) 
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a viantitative course effectiveness rating was obtained (1 = low to 4 = high). 
From the qualitative school observations’, an overall extent of imptementation 
of MBP activities rating was determined (1 = low to 4 = high). When each 
of the three independent variables (principal support, principal course 
attendance, and course effectiveness) were correlated with the dependent Variable 
(extent of MBP implementation), critical information was obtained that was . 
not evident from independent analyses of quantitative and qualitative data. 

Neither the principals’ participation jn the course nor course ef fective- 
ness ratings correlated significantly with the extent of implementation (r = .08 
and .00 pesnactlvelyi, All three factors combined as a single score 
significantly correlated with extent of implementation (r = .576, p£.10). 
This level of significance was not considered by OER sufficient to make pre- 
dictive statements. However, when extent of implementation was correlated 
with avineieat support, the coefficient was ajgniFicant at the .01] probability 
leveT (r= .74). | 
On semi-structured questionnaires, teachers\reported that the most impor- : 
tant factor in whether or not MBP activities were implemented in their class- 
room was their principal's encouragement to use MBP activities. This corrobora- 
ted our initial findings, although there is some overlap because teacher ratings 
of principal support contributed to, but were not the sole criterion, for - 
the principal support variable described above. | “ 
The spiral effect of {nteneating quantitative and qualitative data is . | ‘ 
clearly shown. In Year 1's gathering of quantitative data, no program effects 
“were discerned, yet the evaluators "felt" there was program impact. /In- Year . 
2 and 3 additional gathering of qualitative data in an open-ended, yet syetene z 
atic.manner, provided data not solely obtainable by quantitative means. , 


But level of principal ‘support and extent of implementation could be measured 


“only by integrating quantitatively and qualitatively obtained data. From 


Quantifying Quatitative Data 
these ratings, we discovered the main source of program i eieienetions prin- 
cipal support. Thus, the inadequacy of the quantitative data alone led to 
the integration of quantitative and qualitative data creating new quantifiable 
categories. that. permitted more sophisticated analysis than otherwise possible. 

Through our data-gathering techniques we could probe the context in 
greater depth. Why did principal support vary? Each of the participating 
schools had requested implementation of the MBP and were selected because 
of their strong assurances of support. What happened between the time of 
application to participate in MBP and Years 2-and 3? 

To respond to these questions, we turned again to our case studies for 
Our general knowledge of the school context and MBP implementation. First, from 
‘our principal interviews we discovered that there had been considerable dis- 
satisfaction with the MBP project. director and some artist/instructors during 
Year 1. Aware of these problems, the sponsoring arts organization replaced 
the first MBP project director with a new staff member who they felt could 
better meet the needs of MBP schools. The new director, partly on the basis bs 
of the quantitative course ratings obtained in Year 1, replaced the ineffec- 
tive artist/instructors. In addition, a new staff member was hired to ne 
directly with parents. | 

Our examination of other external forces also helped us to understand 
variations in principal support. The Chicago Public School System, at the 
beginning of Year 1, committed itself to a continuous progress instructional 
program in reading and mathematics. The central office's push to improve 
reading and mathematics skills were interpreted by many principals as a mandate 
to set aside most other aspects of the educational program in order to improve 
reading and math scores. Teac hepaiy too, felt this pressure and interpreted 


any other activities not directly related to math and reading skill building 
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as beyond the reward structure. As a result, MBP activities in many schools 
faced endrmous difficulty in competing for class time with the continuous 
progress program. Teachers and principals in all schools corroborated these 


observations unsoliticated in final interviews, informal conversations, and 


Z 


in the implementation questionnaire. Those experimental principals who showed 
the most concern about the narrowness of the continuous progress focus ana 
" who indicated the necessity of MBP activities to offset this narrowness gave 
the most program support and, in turn, MBP was most successful in thet schools. 
Still another factor affecting principal support was a major change by 
the central administration in faculty.and principal school assignments for 
’ the purpose of ensuring staff racial balance and desegregating faculty. In 
Year 2, two of the ten principals who had worked cooperatively with the MBP 
in the first year were transferred, as were many teachers who were MBP parti- 
| cipants. This meant loss of administrative ‘support and enormods effort by 
MBP staff to rebuild relationships. It also resulted in the loss of carry- 
over by teachers previously involved in MBP couses. 
Thus the quantitative findings alone could not reflect the full impact 
of the’ project on classroom art-related activities. To overcome the personnel 


loss and to offset the negative effects of the continuous progress program 


in implementing art-related activities placed an undue burden on MBP. The 
fact that commitment for continued participation in the project was retained 
by most participants and that art-related activities, were observed despite 
a1] obstacles pointed up the vitality of the project. To sustain itself in 
spite of these enormous negative forces gave additional 'meaning to the quanti- 
tative findings. 
The comparison groups were both adversely affected by these external forces. 
« Classroom learning activities in the "model" school over the three year period 


* tended to be less creative, and the faculty less cohesive. The comparison ” 


school on the other énd of the continuum was further bogged down 


ve | 
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by inertia. The students in both comparison schools indicated less positive © ; 


perceptions of their learning environment over the three-year period. 


uantifying Qualitative Data: The Best of Both Worlds , 


>} 


° ‘ 
We have presented above one detailed example of how quantitative and 


qualitative data can build upon one another in program evaluation. In: fact, 
within the same program, ‘using the same techniques, we were also able to 
discover the attributes of effective artist/instructors. We have also used 
simifar techniques in other program and curriculum evaluations. The importance 
of our findings, we feel, are not specific to the one example presented here. 
On the contrary, we feel that this example illustrates the futility of the 
— quantitative versus qualitative dichotomy. . / 
Both quantitative and qualitative data have inherent limitations. It 
makes sense that an integration of the two will extend the scope of the 
data base. The spiraling effect of integrating the two types of data can 
generate new variables. By abandoning our view of quantitative and quer) i 
tative data as dicotomous, we can work on developing synthesizing methods 


for discerning program effects in an organized, rational, scientific manner. 
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